Method of Determining Video Quality

ABSTRACT

A method and a device utilizing an algorithm using measurement data derived from parameters related to a video-streaming player and/or parameters related to data transport is disclosed. The data are used as input data in a model designed to generate a value corresponding to the quality of the multimedia sequence, such as for example a MOS score.

TECHNICAL FIELD

The present invention relates to a method and a device for determiningthe quality of a multimedia sequence.

BACKGROUND

New high performance radio networks have paved the way for new mobileservices and applications. For example, many of the new services aim atenhancing the experience of a phone conversation; other services providefor transmission of video signals such as video-on-demand and othersimilar multimedia services. All such new services involving receptionof any type of multimedia at a receiver require monitoring of theperceived received quality. This to ensure that users experience goodquality and get the service they expect. In other words multimediaservices such as video streaming quality perceived by the end user isone important service quality measurement for operators of all types ofnetworks. Possible service problems need to be troubleshot.

The subjectively perceived video quality can be estimated with anobjective video quality model. The video quality value can be a MOS(Mean Opinion Score) value or another suitable measure. MOS is the meanvalue of grades from a subjective test, where test persons grade amultimedia clip in a scale ranging from 1 for the poorest quality to 5for the best quality.

Today, there exist a number of commercial tools for measuring andestimating multimedia quality. However, the products for determiningvideo quality on the market today base their video quality estimationmostly on video image analysis. An approach based on video imageanalysis puts a high demand on the computational capacity of the toolused because the algorithms used in such analysis are in themselves verycomputationally demanding. This in turn makes it difficult to produce anoutput result in real-time.

Another problem encountered with existing solutions is that they aresensitive to change in context in the video signal because theytypically base their output on an analysis between different frames of avideo sequence in the case of an estimation of video quality.

Yet another problem with existing solutions, wherein original video iscompared to a received video (so called full reference method), is theneed for synchronization between consecutive frames in order to generatea meaningful output result.

There is therefore a need for a method that requires less computationalpower and which hence can be implemented in a device that is inexpensiveto manufacture and easy to operate.

SUMMARY

It is an object of the present invention to overcome at least some ofthe problems associated with existing methods and tools for determiningthe quality of a multimedia sequence.

This object and others are obtained by a method and a device utilizingan algorithm using measurement data derived from input parametersrelated to a video-streaming player and/or parameters related to datatransport. The multimedia sequence can be transmitted over any type ofnetwork, for example a radio network or a fixed network. The data areused as input data in a model designed to generate a value correspondingto the quality of the multimedia sequence, such as for example a MOSscore. The method in accordance with the invention can advantageously beimplemented by computer software in a computer program product.

Such an approach is significantly faster than methods based on videoimage analysis because the calculations required are much simpler.Typically they do not require any transformation into the frequencydomain and the like.

Moreover, the exact synchronization needed for solutions requiring areference as is the case for solutions based on a full reference videoimage analysis is not necessary, since a solution based on networkparameters does not require a reference. Also, the calculated output,e.g. a MOS value, is not affected if the content is switched as would bethe case in a model using image analysis.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will now be described in more detail by way ofnon-limiting examples and with reference to the accompanying drawings,in which:

FIG. 1 is a general view of a system for streaming video

FIG. 2 is a closer view of some parts of the system shown in FIG. 1, and

FIG. 3 is a flowchart illustrating different steps performed in a devicefor determining perceived multimedia quality

DETAILED DESCRIPTION

In FIG. 1, a general view of a system for measuring the received qualityof a multimedia sequence in accordance with the present invention isdepicted. The multimedia sequence is in the following example a videoclip with associated audio. The system comprises a video streamingserver 101. The video streaming server 101 is the source of a videosequence with associated audio transmitted over a network 103 as a videosignal. The network 103 can for example be a radio network or any othernetwork.

The video sequence transmitted over the network 103 is received by amobile test system 105. The mobile test system 105 typically comprises avideo streaming player 107 and a device 109 including a video qualitymodel. The device 109 utilizes data obtained from the video streamingplayer 107 and from the transport layer in order to generate a valuethat is an estimation of a current end-user perceived video, audio ortotal quality.

In FIG. 2, the mobile test system 105 is shown in more detail. The testsystem 105 is designed to upon reception of a video sequence estimateperceived video quality for video in a computational efficient way. Theestimation is done by, in real-time, collecting measurement data from avideo-streaming player (video streaming), and/or from data transport, inparticular IP layer and above. The data is input and combined in a videoquality estimation model and a video quality value is calculated (forexample a MOS score).

The calculated output (MOS value) is an estimation of end-user perceivedvideo and audio quality. The calculated MOS value is intended to beclose to the result of a subjective test, where test persons view andgrade a video clip. The total streaming quality (MOS value) is expressedas a function of the input variables. Both a linear and non-linear modelcan be used to estimate the MOS value. Also, a combination of linear andnon-linear functions may be employed.

The model is then designed to predict one or many of:

Total quality

Video quality

Audio quality

The perhaps most common video quality problems for mobile videostreaming are high compression (due to low bandwidth over the network),video quality degradation due to packet loss, long initial buffering andre-buffering in the middle of a video. In a preferred embodiment themodel takes into account all of these quality problems and the output,for example presented as a MOS value, then reflects the perceived videoquality.

The input to the model includes a number of these measurementparameters. For example one or many of the following parameters may beutilized:

Audio Codec

Video Codec

Total coded bit rate

Video bit rate

Audio bit rate

Video frame rate

Packet loss rate

Length of initial buffering (as absolute or relative percent of videoclip time)

Number of re-buffering events

Re-buffering frequency (similar to number of re-buffering events)

Start time of re-buffering event (as absolute time or relative percentof total clip time)

Length of re-buffering (as absolute time or relative percent of totalclip time)

Data throughput

IP packet jitter

IP packet size

RTP/RTCP signaling

Buffer size of video client

In different scenarios different parameters may prove to be moreimportant than others when determining a video quality value. To give anexample, in some applications the parameters corresponding to, thepacket loss, the length of initial buffering, and the length ofre-buffering in combination with information regarding the codec and thetotal coded bit rate have proven to give a good estimate of the videoquality value. In such a scenario only a small sub-set of parameters arenecessary in order to generate a good estimate of the current qualityvalue. This is advantageous because the model can then be made simplerand hence the implementation in a device for generating a quality valuecan be faster and less expensive.

If one or many of the input parameters important to the situation athand are not available, other parameters can be used to calculate theones missing. Thus, if for example “Total coded bit rate” is notavailable, “Data throughput”, “IP packet size” and “Buffer size ofclient” can be used to estimate the total coded bit rate. Bufferinginput parameters, such as “number of re-bufferings”, “re-bufferingfrequency”, “start time of re-buffering”, “Length of initial buffering”and “length of re-buffering”, can in the same manner be estimated basedon total coded bit rate, throughput and knowledge about the multimediaplayer's buffer size. Other relationships may be used to estimate otherimportant parameters. A combination effect of many parameters can beused to estimate the quality. For example, the combination effect ofsimultaneous packet loss and re-buffering can be used. The quality isnot necessary a simple addition of quality degradation for packet lossand re-buffering. In fact, in some application the relationship isconceivably more complex and requires a more complex model in order tocorrectly model the perceived video quality.

The video quality model is tuned based on subjective tests, where testpersons have viewed and graded video clips with long and short initialbuffering and long and short middle buffering packet, various loss ratesand potentially other quality degrading effects. Also the number ofmiddle buffering may be varied to tune the model even better. Other waysto tune the model include tuning with help of other objective models,etc.

In FIG. 3 different steps performed in the video quality modeling deviceas described above are shown. Thus, first in a step 301 the video playerreceives a video signal representing a video sequence. The videosequence may or may not be associated with audio. Next, in a step 303, anumber of data for different parameters relating to the received videosignal and/or video player are transferred to the video quality modelingdevice. The data typically relate to one or many of the parameterslisted above in conjunction to FIG. 2, e.g. Video Codec, Audio Codec,Total coded bit rate, Video bit rate, Audio bit rate. Video frame rate,Packet loss rate, Start time of re-buffering (as absolute time orrelative percent of total clip time), Length of initial buffering (asabsolute or relative percent of video clip time), Number of re-bufferingevents, Re-buffering frequency (similar to number of re-bufferingevents), Length of re-buffering (as absolute time or relative percent oftotal clip time), Data throughput, IP packet jitter, IP packet size,RTP/RTCP signaling, Buffer size of video client or other parametersrelevant for the particular situation as the situation may be.

Thereupon, in a step 305, a value indicative of the quality of thereceived video signal is calculated. Depending on what the device isdesigned to generate as an output the value can be all or a subset oftotal quality, video quality and audio quality. The calculationsperformed in step 305 can differ for different applications andscenarios and will be described more closely below. Finally, in a step307, the device outputs a result. For example, the video quality valuemay be presented as a numerical value such as a MOS. This value is thenused as input data for optimizing the network or similar tasks.

In accordance with one preferred embodiment, the basic function of themodel can be described with:

TOT_MOS_(pred)=func(Qual_(encoding),Qual_(buff),Qual_(pl))

The base quality (quality of the encoding, Qual_(encoding)) can bedescribed with a function:

y=c ₀ -c ₁ ·e ^(−λ·x)

where c₀, c₁ and λ are constants. The constants have different valuesfor different codecs.

Qual_(buff) and Qual_(pl) reduce the output MOS value based on initialbuffering time, re-buffering percentage, re-buffering frequency andpacket loss, respectively.

The packet loss effect for a logging period with packet loss can beexpressed by:

Qual_(PL)=const*(Qual_(encoding)−1)·ξ+1

where the factor ξ is defined as

$\xi = \frac{{PLR}_{u} - {PLR}_{mean}}{{PLR}_{u} - {PLR}_{l}}$

in which, PLR_(u) and PLR_(l) are the upper and lower packet loss ratelimits respectively and PLR_(mean) is the average packet loss rate ofthe current logging window; hence 0≦ξ≦1

${PLR}_{mean} = {\frac{1}{N} \cdot {\sum\limits_{i = 1}^{N}\; {PLR}_{i}}}$

For packet loss a lower and upper limit can be defined, based onsubjective tests. Packet loss rate (PLR) lower than the lower limit arethen counted as non-visible and will hence not affect the MOS value. Atthe other end of the scale, a PLR equal to or above the upper limit areper default set to be equally very bad (worst) quality. Thus, thefollowing restrictions are applied on the instant PLR values:

PLR _(i)=min(PLR _(j) ,PLR _(u)), and

PLR _(i)=max(PLR _(j) ,PLR _(l))

Adding re-buffering and initial buffering effect on quality is then forexample done by:

TOT _(—) MOS _(pred)=Qual_(PL)−Qual_(buff)

and Qual_(buff) is calculated by:

Qual_(buff) =C ₀ +C ₁·INIT_PERC+C ₂·BUF_(—PERC+) C ₃·BUF_FRQ, and

Where the variables are:

BUF_PERC: Re-buffering time/(playing time+rebuffering time+initialbuffering)

INIT_PERC: Initial buffering time/(playing time+rebuffering time+initialbuffering)

BUF_FRQ: Number of rebuffering events per minute

It may be that the effect of the packet loss rate, re-buffering andinitial buffering is better modeled as a function:

TOT_MOS_(pred) =f(Qual_(PL), Qual_(buff)),

rather than a pure linear effect.

To handle severe packet losses and the effect on perceived quality highpacket loss rates are preferably weighted higher than lower rates. Tohandle long re-buffering times the output MOS score may be truncated forvery long re-buffering percentages. For example, 67% or higherre-buffering percent could be modeled to always result in the lowestvideo quality, e.g. MOS=1.

Using the invention a combination of input parameters can be used toestimate the complete video quality. One advantage of the invention isthat only a small subset of parameters for a video sequence are requiredin order to generate a good video quality value. For example, thecombination of interruptions (re-buffering) and packet loss can alone beused to estimate quality in addition with codec and bit-rate.

The invention is computational efficient due to the fact that the videoimages are not analyzed. The input parameters are few; still good videoquality estimation can be obtained. The invention also takes intoaccount the most important video quality degradation factors: codingquality and degradation due to transport errors. Coding qualityestimation uses information about codec. Transport errors areessentially low throughput or packet loss. Low throughput might causelong initial buffering and re-buffering events. Packet losses will causeimage and/or audio distortions and quality degradation due to temporalproblems in the video.

The invention as described herein is, due to the fact that it iscomputational efficient, able to produce a video quality value inreal-time. It is hence very well suited for test situations where ascore must be produced in the moment of measurement.

In addition using the invention as described herein there is no need forexact synchronization as is the case solutions for based on a videoimage analysis, since a solution based on network parameters does notrequire a reference. Also, the calculated output, e.g. a MOS value, isnot affected if the content is switched as would be the case in a modelusing image analysis.

1-19. (canceled)
 20. A mobile test system for measuring the receivedquality of a multimedia sequence, comprising: a video streaming playeroperative to receive a multimedia sequence from a video streaming serveracross a network, and further operative to extract from the multimediasequence quality parameters; a video quality model operative to receivemultimedia sequence quality parameters from the video streaming playerand data transport parameters, and further operative to generatetherefrom a metric of perceived video quality.
 21. The system of claim20, wherein the one or more parameters received by the video qualitymodel comprise parameters corresponding to a codec and a coded bit rateassociated with the multimedia sequence.
 22. The system of claim 22,wherein the video quality model is operative to consider one or more ofa packet loss parameter, a length of initial buffering parameter, and alength of re-buffering parameter, to generate the metric of perceivedvideo quality.
 23. The system of claim 20, wherein the video qualitymodel is operative to generate a Mean Opinion Score (MOS) value as themetric of perceived video quality.
 24. The system of claim 24, whereinthe video quality model is operative to generate the metric of perceivedvideo quality using a function of packet loss rate parameter, are-buffering parameter, and an initial buffering data parameter.
 25. Thesystem of claim 20, wherein the video quality model is further operativeto generate a metric of perceived quality is for one or both of a totalquality and an audio quality.
 26. The system of claim 20, wherein thenetwork is a radio network.
 27. The system of claim 20, wherein thevideo quality model is further operative to derive or estimate one ormore input parameters from other parameters.
 28. A non-transientcomputer program product including software operative to cause a mobiletest system, configured to communicate with a server via a communicationnetwork, to: receive a multimedia sequence; extract one or more inputparameters derived from transmission and/or one or more multimediaplayer parameters associated with the received multimedia sequence;generate a quality value corresponding to a subjective quality scorebased on at least one of the extracted input parameters; and output thequality value as a measure of the quality of the received multimediasequence.
 29. The non-transient computer program product of claim 28wherein the one or more input parameters comprise parameters associatedwith at least one of a video-streaming player and data transport. 30.The non-transient computer program product of claim 28 wherein the oneor more input parameters comprise parameters corresponding to a codecand a coded bit rate associated with the multimedia sequence.
 31. Thenon-transient computer program product of claim 30 wherein, whenexecuted by the mobile test system, the software causes the multimediaquality model device to utilize one or more of a packet loss parameter,a length of initial buffering parameter, and a length of re-bufferingparameter, to generate the quality value.
 32. The non-transient computerprogram product of claim 28 wherein the generated quality valuecorresponds to a Mean Opinion Score (MOS) value.
 33. The non-transientcomputer program product of claim 28 wherein the software causes themultimedia quality model device to calculate the generated quality valueusing a function of packet loss rate parameter, a re-bufferingparameter, and an initial buffering data parameter.
 34. Thenon-transient computer program product of claim 28 wherein the softwarecauses the multimedia quality model device to generate the quality valuefor one or more of a total quality, a video quality, and an audioquality.
 35. The non-transient computer program product of claim 28wherein the software causes the multimedia streaming device to receivethe multimedia sequence over a radio communication network.
 36. Thenon-transient computer program product of claim 28 wherein the softwarecauses the multimedia quality model device to derive or estimate the oneor more input parameters from other parameters.